Vehicle undercarriage imaging

ABSTRACT

Images of an undercarriage of a vehicle may be captured via one or more cameras. A point cloud may be determined based on the images. The point cloud may includes points positioned in a virtual three-dimensional space. A stitched image may be determined based on the point cloud by projecting the point cloud onto a virtual camera view. The stitched image may be stored on a storage device

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. 120 to U.S.patent application Ser. No. 17/190,268 (Atty Docket No. FYSNP072),titled “VEHICLE UNDERCARRIAGE IMAGING,” filed Mar. 2, 2021, by Holzer etal., which is hereby incorporated by reference in its entirety and forall purposes.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the United States Patent andTrademark Office patent file or records but otherwise reserves allcopyright rights whatsoever.

TECHNICAL FIELD

The present disclosure relates generally to the capture and presentationof image data of an object, and more specifically to image data of avehicle undercarriage.

DESCRIPTION OF RELATED ART

Vehicles need to be inspected for damage on different occasions. Forexample, a vehicle may be inspected after an accident to evaluate orsupport an insurance claim or police report. As another example, avehicle may be inspected before and after the rental of a vehicle, orbefore buying or selling a vehicle.

Vehicle inspection using conventional approaches is a largely manualprocess. Typically, a person walks around the vehicle and manually notesdamage and conditions. This process is time-intensive, resulting insignificant costs. The manual inspection results also vary based on theperson. For example, a person may be more or less experienced inevaluating damage. The variation in results can yield a lack of trustand potential financial losses, for example when buying and sellingvehicles or when evaluating insurance claims.

BRIEF DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 illustrates a method for damage detection, performed inaccordance with one or more embodiments.

FIG. 2 illustrates a method of damage detection data capture, performedin accordance with one or more embodiments.

FIG. 3 illustrates a method for component-level damage detection,performed in accordance with one or more embodiments.

FIG. 4 illustrates an object-level damage detection method, performed inaccordance with one or more embodiments.

FIG. 5 illustrates a computer system that can be used in accordance withone or more embodiments.

FIG. 6 shows a top-down diagram of a damage detection portal arranged asa gate, configured in accordance with one or more embodiments.

FIG. 7 shows a perspective diagram of a damage detection portal arrangedas a gate, configured in accordance with one or more embodiments.

FIG. 8 shows a top-down diagram of a damage detection portal arranged asa tunnel, configured in accordance with one or more embodiments.

FIG. 9 shows a perspective diagram of a damage detection portal arrangedas a tunnel, configured in accordance with one or more embodiments.

FIGS. 10A and 10B show top-down views of a damage detection portal,configured in accordance with one or more embodiments.

FIGS. 11A and 11B show a top-down views of a damage detection portal,configured in accordance with one or more embodiments.

FIG. 12 shows a side-view diagram of a damage detection portal as avehicle drives through it, configured in accordance with one or moreembodiments.

FIG. 13 illustrates the capture of image data via a damage detectionportal, configured in accordance with one or more embodiments.

FIGS. 14-16 show simulated images generated from a damage detectionportal, configured in accordance with one or more embodiments.

FIGS. 17-19 show images captured via a damage detection portal andpresented in a user interface, generated in accordance with one or moreembodiments.

FIG. 20 illustrate a portion of the user interface in which detecteddamage is shown, configured in accordance with one or more embodiments.

FIGS. 21-25 illustrate images captured via the damage detection portaland presented in the user interface, configured in accordance with oneor more embodiments.

FIG. 26 illustrates an overview method for the operation of a damagedetection portal, configured in accordance with one or more embodiments.

FIG. 27 shows a side-view of a damage detection portal as a vehicledrives through it, configured in accordance with one or moreembodiments.

FIG. 28 illustrates an image extraction method, performed in accordancewith one or more embodiments.

FIG. 29 describes a method for calibrating a damage detection portal,performed in accordance with one or more embodiments.

FIG. 30 illustrates a method for damage detection, performed inaccordance with one or more embodiments.

FIG. 31 illustrates a method for localized MVIDMR generation, performedin accordance with one or more embodiments.

FIGS. 32-35 illustrate examples of a semantic mesh, generated inaccordance with one or more embodiments.

FIGS. 36-41 illustrate examples of a component analysis applied to thesemantic mesh, performed in accordance with one or more embodiments.

FIG. 42 illustrates an overview method, performed in accordance with oneor more embodiments.

FIG. 43 illustrates an undercarriage imaging system, configured inaccordance with one or more embodiments.

FIG. 44 illustrates an image captured by a single camera in a mirrorconfiguration, generated in accordance with one or more embodiments.

FIGS. 45A and 45B illustrate images generated by an undercarriageimaging system configured in accordance with one or more embodiments.

FIGS. 46A, 46B, and 46C illustrate stitched images captured of theundercarriage of a vehicle by undercarriage capture systems configuredin accordance with one or more embodiments.

FIGS. 47A and 47B illustrate images generated by an undercarriageimaging system configured in accordance with one or more embodiments.

FIG. 48 illustrates a method A for stitching an undercarriage image,performed in accordance with one or more embodiments.

FIG. 49 illustrates a method B for stitching an undercarriage image,performed in accordance with one or more embodiments.

FIG. 50 illustrates a method performed in accordance with one or moreembodiments.

FIGS. 51 and 52 show examples of presenting multiple MVIDMRs, providedin accordance with one or more embodiments.

FIG. 53 shows the determination of a correspondence between imagescaptured by the undercarriage system in accordance with one or moreembodiments.

FIG. 54 shows another example of presenting multiple MVIDMRs inaccordance with one or more embodiments.

FIGS. 55A and 55B illustrate examples of configurations of anundercarriage imaging system with multiple mirrors, arranged inaccordance with one or more embodiments.

TECHNICAL DESCRIPTION

The undercarriage of a vehicle often includes various components thatare vital to the vehicle's continued operation and performance. However,inspecting a vehicle's undercarriage using conventional techniques is adifficult task. Typically a vehicle must be driven over a bay in which ahuman can move beneath the vehicle and manually inspect the vehicle'sundercarriage for wear and damage.

Conventional techniques for providing visual data present specificchallenges when applied to the undercarriage of a vehicle. A vehicle'sundercarriage is typically relatively close to the ground, which meansthat a traditional camera would have a limited field of view even if itwere positioned beneath the vehicle. Moreover, different vehicles havedifferent ground clearances, so conventional techniques do not providefor reliably capturing and interpreting image data of differentvehicle's undercarriages in an automated fashion. Also, for manyvehicles the undercarriage is an assortment of three-dimensional parts,and not a relatively flat surface. Conventional techniques typicallyfail to capture this dimensionality.

Conventional techniques for providing visual data of a vehicleundercarriage require significantly more ground clearance than istypically available. For example, a vehicle may be driven over a ramp oropen bay in which a camera is positioned. However, many such approachesrequire modifications to existing infrastructure, such as digging a holein a roadway. Other such approaches require careful vehicle navigation,such as driving the vehicle over a ramp with an open middle component.

According to various embodiments, techniques and mechanisms describedherein provide for capturing and processing visual data associated witha vehicle's undercarriage. The visual data may be used to detect damageto the undercarriage or otherwise present the undercarriage for visualinspection. Moreover, techniques and mechanisms described herein mayfunction in a configuration where one or more cameras are positioneddirectly on the ground and then driven over by the vehicle.

In some embodiments, image data may be presented in a way that stichestogether multiple images of the vehicle's undercarriage. Alternatively,or additionally, an MVIDMR of a vehicle's undercarriage may beconstructed. The MVIDMR may present a view of the vehicle'sundercarriage that may be navigated in one or more dimensions.

According to various embodiments, techniques and mechanisms describedherein may be used to identify and represent damage to an object such asa vehicle. The damage detection techniques may be employed by untrainedindividuals. For example, an individual may collect multi-view data ofan object, and the system may detect the damage automatically.

According to various embodiments, various types of damage may bedetected. For a vehicle, such data may include, but is not limited to:scratches, dents, flat tires, cracked glass, broken glass, or other suchdamage.

In some implementations, a user may be guided to collect multi-view datain a manner that reflects the damage detection process. For example,when the system detects that damage may be present, the system may guidethe user to take additional images of the portion of the object that isdamaged.

According to various embodiments, techniques and mechanisms describedherein may be used to create damage estimates that are consistent overmultiple captures. In this way, damage estimates may be constructed in amanner that is independent of the individual wielding the camera anddoes not depend on the individual's expertise. In this way, the systemcan automatically detect damage in an instant, without requiring humanintervention.

Although various techniques and mechanisms are described herein by wayof example with reference to detecting damage to vehicles, thesetechniques and mechanisms are widely applicable to detecting damage to arange of objects. Such objects may include, but are not limited to:houses, apartments, hotel rooms, real property, personal property,equipment, jewelry, furniture, offices, people, and animals.

FIG. 1 illustrates a method 100 for damage detection, performed inaccordance with one or more embodiments. According to variousembodiments, the method 100 may be performed at a damage detectionportal or computing device in communication with a damage detectionportal. Alternately, or additionally, some or all of the method 100 maybe performed at a remote computing device such as a server. The method100 may be used to detect damage to any of various types of objects.However, for the purpose of illustration, many examples discussed hereinwill be described with reference to vehicles.

At 102, multi-view data of an object is captured. According to variousembodiments, the multi-view data may include images captured fromdifferent viewpoints. For example, a user may walk around a vehicle andcapture images from different angles. In some configurations, themulti-view data may include data from various types of sensors. Forexample, the multi-view data may include data from more than one camera.As another example, the multi-view data may include data from a depthsensor. As another example, the multi-view data may include datacollected from an inertial measurement unit (IMU). IMU data may includeposition information, acceleration information, rotation information, orother such data collected from one or more accelerometers or gyroscopes.

In particular embodiments, the multi-view data may be aggregated toconstruct a multi-view representation. Additional details regardingmulti-view data and damage detection are discussed in co-pending andcommonly assigned U.S. patent application Ser. No. 16/692,133, “DAMAGEDETECTION FROM MULTI-VIEW VISUAL DATA”, by Holzer et al., filed Nov. 22,2019, which is hereby incorporated by reference in its entirety and forall purposes.

At 104, damage to the object is detected based on the capturedmulti-view data. In some implementations, the damage may be detected byevaluating some or all of the multi-view data with a neural network, bycomparing some or all of the multi-view data with reference data, and/orany other relevant operations for damage detection. Additional detailsregarding damage detection are discussed throughout the application.

At 106, a representation of the detected damage is stored on a storagemedium or transmitted via a network. According to various embodiments,the representation may include some or all of a variety of information.For example, the representation may include an estimated dollar value.As another example, the representation may include a visual depiction ofthe damage. As still another example, a list of damaged parts may beprovided. Alternatively, or additionally, the damaged parts may behighlighted in a 3D CAD model.

In some embodiments, a visual depiction of the damage may include animage of actual damage. For example, once the damage is identified at104, one or more portions of the multi-view data that include images ofthe damaged portion of the object may be selected and/or cropped.

In some implementations, a visual depiction of the damage may include anabstract rendering of the damage. An abstract rendering may include aheatmap that shows the probability and/or severity of damage using acolor scale. Alternatively, or additionally, an abstract rendering mayrepresent damage using a top-down view or other transformation. Bypresenting damage on a visual transformation of the object, damage (orlack thereof) to different sides of the object may be presented in astandardized manner.

FIG. 2 illustrates a method 200 of damage detection data capture,performed in accordance with one or more embodiments. According tovarious embodiments, the method 200 may be performed at a damagedetection portal or computing device in communication with a damagedetection portal. The method 200 may be used to detect damage to any ofvarious types of objects. However, for the purpose of illustration, manyexamples discussed herein will be described with reference to vehicles.

A request to capture input data for damage detection for an object isreceived at 202. In some implementations, the request to capture inputdata may be received at a damage detection portal or computing device incommunication with a damage detection portal. In particular embodiments,the object may be a vehicle such as a car, truck, or sports utilityvehicle.

An object model for damage detection is determined at 204. According tovarious embodiments, the object model may include reference data for usein evaluating damage and/or collecting images of an object. For example,the object model may include one or more reference images of similarobjects for comparison. As another example, the object model may includea trained neural network. As yet another example, the object model mayinclude one or more reference images of the same object captured at anearlier point in time. As yet another example, the object model mayinclude a 3D model (such as a CAD model) or a 3D mesh reconstruction ofthe corresponding vehicle.

In some embodiments, the object model may be determined based on userinput. For example, the user may identify a vehicle in general or a car,truck, or sports utility vehicle in particular as the object type.

In some implementations, the object model may be determinedautomatically based on data captured as part of the method 200. In thiscase, the object model may be determined after the capturing of one ormore images at 206.

At 206, an image of the object is captured. According to variousembodiments, capturing the image of the object may involve receivingdata from one or more of various sensors. Such sensors may include, butare not limited to, one or more cameras, depth sensors, accelerometers,and/or gyroscopes. The sensor data may include, but is not limited to,visual data, motion data, and/or orientation data. In someconfigurations, more than one image of the object may be captured.Alternatively, or additionally, video footage may be captured.

According to various embodiments, a camera or other sensor located at acomputing device may be communicably coupled with the computing devicein any of various ways. For example, in the case of a mobile phone orlaptop, the camera may be physically located within the computingdevice. As another example, in some configurations a camera or othersensor may be connected to the computing device via a cable. As stillanother example, a camera or other sensor may be in communication withthe computing device via a wired or wireless communication link.

According to various embodiments, as used herein the term “depth sensor”may be used to refer to any of a variety of sensor types that may beused to determine depth information. For example, a depth sensor mayinclude a projector and camera operating in infrared light frequencies.As another example, a depth sensor may include a projector and cameraoperating in visible light frequencies. For instance, a line-laser orlight pattern projector may project a visible light pattern onto anobject or surface, which may then be detected by a visible light camera.

One or more features of the captured image or images are extracted at208. In some implementations, extracting one or more features of theobject may involve constructing a multi-view capture that presents theobject from different viewpoints. If a multi-view capture has alreadybeen constructed, then the multi-view capture may be updated based onthe new image or images captured at 206. Alternatively, or additionally,feature extraction may involve performing one or more operations such asobject recognition, component identification, orientation detection, orother such steps.

At 210, the extracted features are compared with the object model.According to various embodiments, comparing the extracted features tothe object model may involve making any comparison suitable fordetermining whether the captured image or images are sufficient forperforming damage comparison. Such operations may include, but are notlimited to: applying a neural network to the captured image or images,comparing the captured image or images to one or more reference images,and/or performing any of the operations discussed with respect to FIGS.3 and 4 .

A determination is made at 212 as to whether to capture an additionalimage of the object. In some implementations, the determination may bemade at least in part based on an analysis of the one or more imagesthat have already been captured.

In some embodiments, a preliminary damage analysis may be implementedusing as input the one or more images that have been captured. If thedamage analysis is inconclusive, then an additional image may becaptured. Techniques for conducting damage analysis are discussed inadditional detail with respect to the methods 300 and 400 shown in FIGS.3 and 4 .

In some embodiments, the system may analyze the captured image or imagesto determine whether a sufficient portion of the object has beencaptured in sufficient detail to support damage analysis. For example,the system may analyze the capture image or images to determine whetherthe object is depicted from all sides. As another example, the systemmay analyze the capture image or images to determine whether each panelor portion of the object is shown in a sufficient amount of detail. Asyet another example, the system may analyze the capture image or imagesto determine whether each panel or portion of the object is shown from asufficient number of viewpoints.

If the determination is made to capture an additional image, then at 214image collection guidance for capturing the additional image isdetermined. In some implementations, the image collection guidance mayinclude any suitable instructions for capturing an additional image thatmay assist in changing the determination made at 212. Such guidance mayinclude an indication to capture an additional image from a targetedviewpoint, to capture an additional image of a designated portion of theobject, or to capture an additional image at a different level ofclarity or detail. For example, if possible damage is detected, thenfeedback may be provided to capture additional detail at the damagedlocation.

At 216, image collection feedback is provided. According to variousembodiments, the image collection feedback may include any suitableinstructions or information for assisting a user in collectingadditional images. Such guidance may include, but is not limited to,instructions to collect an image at a targeted camera position,orientation, or zoom level. Alternatively, or additionally, a user maybe presented with instructions to capture a designated number of imagesor an image of a designated portion of the object.

For example, a user may be presented with a graphical guide to assistthe user in capturing an additional image from a target perspective. Asanother example, a user may be presented with written or verbalinstructions to guide the user in capturing an additional image.

When it is determined to not capture an additional image of the object,then at 218 the captured image or images are stored. In someimplementations, the captured images may be stored on a storage deviceand used to perform damage detection, as discussed with respect to themethods 300 and 400 in FIGS. 3 and 4 . Alternatively, or additionally,the images may be transmitted to a remote location via a networkinterface.

FIG. 3 illustrates a method 300 for component-level damage detection,performed in accordance with one or more embodiments. According tovarious embodiments, the method 300 may be performed at a damagedetection portal or computing device in communication with a damagedetection portal. The method 300 may be used to detect damage to any ofvarious types of objects. However, for the purpose of illustration, manyexamples discussed herein will be described with reference to vehicles.

A skeleton is extracted from input data at 302. According to variousembodiments, the input data may include visual data collected asdiscussed with respect to the method 300 shown in FIG. 3 .Alternatively, or additionally, the input data may include previouslycollected visual data, such as visual data collected without the use ofrecording guidance.

In some implementations, the input data may include one or more imagesof the object captured from different perspectives. Alternatively, oradditionally, the input data may include video data of the object. Inaddition to visual data, the input data may also include other types ofdata, such as IMU data.

According to various embodiments, skeleton detection may involve one ormore of a variety of techniques. Such techniques may include, but arenot limited to: 2D skeleton detection using machine learning, 3D poseestimation, and 3D reconstruction of a skeleton from one or more 2Dskeletons and/or poses.

Calibration image data associated with the object is identified at 304.According to various embodiments, the calibration image data may includeone or more reference images of similar objects or of the same object atan earlier point in time. Alternatively, or additionally, thecalibration image data may include a neural network used to identifydamage to the object.

A skeleton component is selected for damage detection at 306. In someimplementations, a skeleton component may represent a panel of theobject. In the case of a vehicle, for example, a skeleton component mayrepresent a door panel, a window, or a headlight. Skeleton componentsmay be selected in any suitable order, such as sequentially, randomly,in parallel, or by location on the object.

According to various embodiments, when a skeleton component is selectedfor damage detection, a multi-view capture of the skeleton component maybe constructed. Constructing a multi-view capture of the skeletoncomponent may involve identifying different images in the input datathat capture the skeleton component from different viewpoints. Theidentified images may then be selected, cropped, and combined to producea multi-view capture specific to the skeleton component.

A viewpoint of the skeleton component is selected for damage detectionat 304. In some implementations, each viewpoint included in themulti-view capture of the skeleton component may be analyzedindependently. Alternatively, or additionally, more than one viewpointmay be analyzed simultaneously, for instance by providing the differentviewpoints as input data to a machine learning model trained to identifydamage to the object. In particular embodiments, the input data mayinclude other types of data, such as 3D visual data or data capturedusing a depth sensor or other type of sensor.

According to various embodiments, one or more alternatives to skeletonanalysis at 302-310 may be used. For example, an object part (e.g.,vehicle component) detector may be used to directly estimate the objectparts. As another example, an algorithm such as a neural network may beused to map an input image to a top-down view of an object such as avehicle (and vice versa) in which the components are defined. As yetanother example, an algorithm such as a neural network that classifiesthe pixels of an input image as a specific component can be used toidentify the components. As still another example, component-leveldetectors may be used to identify specific components of the object. Asyet another alternative, a 3D reconstruction of the vehicle may becomputed and a component classification algorithm may be run on that 3Dmodel. The resulting classification can then be back-projected into eachimage. As still another alternative, a 3D reconstruction of the vehiclecan be computed and fitted to an existing 3D CAD model of the vehicle inorder to identify the single components.

At 310, the calibration image data is compared with the selectedviewpoint to detect damage to the selected skeleton component. Accordingto various embodiments, the comparison may involve applying a neuralnetwork to the input data. Alternatively, or additionally, an imagecomparison between the selected viewpoint and one or more referenceimages of the object captured at an earlier point in time may beperformed.

A determination is made at 312 as to whether to select an additionalviewpoint for analysis. According to various embodiments, additionalviewpoints may be selected until all available viewpoints are analyzed.Alternatively, viewpoints may be selected until the probability ofdamage to the selected skeleton component has been identified to adesignated degree of certainty.

Damage detection results for the selected skeleton component areaggregated at 314. According to various embodiments, damage detectionresults from different viewpoints to a single damage detection resultper panel resulting in a damage result for the skeleton component. Forexample, a heatmap may be created that shows the probability and/orseverity of damage to a vehicle panel such as a vehicle door. Accordingto various embodiments, various types of aggregation approaches may beused. For example, results determined at 310 for different viewpointsmay be averaged. As another example, different results may be used to“vote” on a common representation such as a top-down view. Then, damagemay be reported if the votes are sufficiently consistent for the panelor object portion.

A determination is made at 316 as to whether to select an additionalskeleton component for analysis. In some implementations, additionalskeleton components may be selected until all available skeletoncomponents are analyzed.

Damage detection results for the object are aggregated at 314. Accordingto various embodiments, damage detection results for differentcomponents may be aggregated into a single damage detection result forthe object as a whole. For example, creating the aggregated damageresults may involve creating a top-down view. As another example,creating the aggregated damage results may involve identifyingstandardized or appropriate viewpoints of portions of the objectidentified as damaged. As yet another example, creating the aggregateddamage results may involve tagging damaged portions in a multi-viewrepresentation. As still another example, creating the aggregated damageresults may involve overlaying a heatmap on a multi-view representation.As yet another example, creating the aggregated damage results mayinvolve selecting affected parts and presenting them to the user.Presenting may be done as a list, as highlighted elements in a 3D CADmodel, or in any other suitable fashion.

In particular embodiments, techniques and mechanisms described hereinmay involve a human to provide additional input. For example, a humanmay review damage results, resolve inconclusive damage detectionresults, or select damage result images to include in a presentationview. As another example, human review may be used to train one or moreneural networks to ensure that the results computed are correct and areadjusted as necessary.

FIG. 4 illustrates an object-level damage detection method 400,performed in accordance with one or more embodiments. The method 400 maybe performed at a damage detection portal or computing device incommunication with a damage detection portal. The method 400 may be usedto detect damage to any of various types of objects.

Evaluation image data associated with the object is identified at 402.According to various embodiments, the evaluation image data may includesingle images captured from different viewpoints. As discussed herein,the single images may be aggregated into a multi-view capture, which mayinclude data other than images, such as IMU data.

An object model associated with the object is identified at 404. In someimplementations, the object model may include a 2D or 3D standardizedmesh, model, or abstracted representation of the object. For instance,the evaluation image data may be analyzed to determine the type ofobject that is represented. Then, a standardized model for that type ofobject may be retrieved. Alternatively, or additionally, a user mayselect an object type or object model to use. The object model mayinclude a top-down view of the object.

Calibration image data associated with the object is identified at 406.According to various embodiments, the calibration image data may includeone or more reference images. The reference images may include one ormore images of the object captured at an earlier point in time.Alternatively, or additionally, the reference images may include one ormore images of similar objects. For example, a reference image mayinclude an image of the same type of car as the car in the images beinganalyzed.

In some implementations, the calibration image data may include a neuralnetwork trained to identify damage. For instance, the calibration imagedata may be trained to analyze damage from the type of visual dataincluded in the evaluation data.

The calibration data is mapped to the object model at 408. In someimplementations, mapping the calibration data to the object model mayinvolve mapping a perspective view of an object from the calibrationimages to a top-down view of the object.

The evaluation image data is mapped to the object model at 410. In someimplementations, mapping the evaluation image data to the object modelmay involve determine a pixel-by-pixel correspondence between the pixelsof the image data and the points in the object model. Performing such amapping may involve determining the camera position and orientation foran image from IMU data associated with the image.

In some embodiments, a dense per-pixel mapping between an image and thetop-down view may be estimated at 410. Alternatively, or additionally,location of center of an image may be estimated with respect to thetop-down view. For example, a machine learning algorithm such as deepnet may be used to map the image pixels to coordinates in the top-downview. As another example, joints of a 3D skeleton of the object may beestimated and used to define the mapping. As yet another example,component-level detectors may be used to identify specific components ofthe object.

In some embodiments, the location of one or more object parts within theimage may be estimated. Those locations may then be used to map datafrom the images to the top-down view. For example, object parts may beclassified on a pixel-wise basis. As another example, the centerlocation of object parts may be determined. As another example, thejoints of a 3D skeleton of an object may be estimated and used to definethe mapping. As yet another example, component-level detectors may beused for specific object components.

In some implementations, images may be mapped in a batch via a neuralnetwork. For example, a neural network may receive as input a set ofimages of an object captured from different perspectives. The neuralnetwork may then detect damage to the object as a whole based on the setof input images.

The mapped evaluation image data is compared to the mapped calibrationimage data at 412 to identify any differences. According to variousembodiments, the data may be compared by running a neural network on amulti-view representation as a whole. Alternatively, or additional, theevaluation and image data may be compared on an image-by-image basis.

If it is determined at 414 that differences are identified, then at 416a representation of the identified differences is determined. Accordingto various embodiments, the representation of the identified differencesmay involve a heatmap of the object as a whole. For example, a heatmapof a top-down view of a vehicle showing damage is illustrated in FIG. 2. Alternatively, one or more components that are damaged may be isolatedand presented individually.

At 418, a representation of the detected damage is stored on a storagemedium or transmitted via a network. In some implementations, therepresentation may include an estimated dollar value. Alternatively, oradditionally, the representation may include a visual depiction of thedamage. Alternatively, or additionally, affected parts may be presentedas a list and/or highlighted in a 3D CAD model.

In particular embodiments, damage detection of an overall objectrepresentation may be combined with damage representation on one or morecomponents of the object. For example, damage detection may be performedon a closeup of a component if an initial damage estimation indicatesthat damage to the component is likely.

With reference to FIG. 5 , shown is a particular example of a computersystem that can be used to implement particular examples. For instance,the computer system 500 can be used to provide MVIDMRs according tovarious embodiments described above. According to various embodiments, asystem 500 suitable for implementing particular embodiments includes aprocessor 501, a memory 503, an interface 511, and a bus 515 (e.g., aPCI bus).

The system 500 can include one or more sensors 509, such as lightsensors, accelerometers, gyroscopes, microphones, cameras includingstereoscopic or structured light cameras. As described above, theaccelerometers and gyroscopes may be incorporated in an IMU. The sensorscan be used to detect movement of a device and determine a position ofthe device. Further, the sensors can be used to provide inputs into thesystem. For example, a microphone can be used to detect a sound or inputa voice command.

In the instance of the sensors including one or more cameras, the camerasystem can be configured to output native video data as a live videofeed. The live video feed can be augmented and then output to a display,such as a display on a mobile device. The native video can include aseries of frames as a function of time. The frame rate is oftendescribed as frames per second (fps). Each video frame can be an arrayof pixels with color or gray scale values for each pixel. For example, apixel array size can be 512 by 512 pixels with three color values (red,green and blue) per pixel. The three color values can be represented byvarying amounts of bits, such as 24, 30, 5, 40 bits, etc. per pixel.When more bits are assigned to representing the RGB color values foreach pixel, a larger number of colors values are possible. However, thedata associated with each image also increases. The number of possiblecolors can be referred to as the color depth.

The video frames in the live video feed can be communicated to an imageprocessing system that includes hardware and software components. Theimage processing system can include non-persistent memory, such asrandom-access memory (RAM) and video RAM (VRAM). In addition,processors, such as central processing units (CPUs) and graphicalprocessing units (GPUs) for operating on video data and communicationbusses and interfaces for transporting video data can be provided.Further, hardware and/or software for performing transformations on thevideo data in a live video feed can be provided.

In particular embodiments, the video transformation components caninclude specialized hardware elements configured to perform functionsnecessary to generate a synthetic image derived from the native videodata and then augmented with virtual data. In data encryption,specialized hardware elements can be used to perform a specific datatransformation, i.e., data encryption associated with a specificalgorithm. In a similar manner, specialized hardware elements can beprovided to perform all or a portion of a specific video datatransformation. These video transformation components can be separatefrom the GPU(s), which are specialized hardware elements configured toperform graphical operations. All or a portion of the specifictransformation on a video frame can also be performed using softwareexecuted by the CPU.

The processing system can be configured to receive a video frame withfirst RGB values at each pixel location and apply operation to determinesecond RGB values at each pixel location. The second RGB values can beassociated with a transformed video frame which includes synthetic data.After the synthetic image is generated, the native video frame and/orthe synthetic image can be sent to a persistent memory, such as a flashmemory or a hard drive, for storage. In addition, the synthetic imageand/or native video data can be sent to a frame buffer for output on adisplay or displays associated with an output interface. For example,the display can be the display on a mobile device or a view finder on acamera.

In general, the video transformations used to generate synthetic imagescan be applied to the native video data at its native resolution or at adifferent resolution. For example, the native video data can be a 512 by512 array with RGB values represented by 24 bits and at frame rate of 24fps. In some embodiments, the video transformation can involve operatingon the video data in its native resolution and outputting thetransformed video data at the native frame rate at its nativeresolution.

In other embodiments, to speed up the process, the video transformationsmay involve operating on video data and outputting transformed videodata at resolutions, color depths and/or frame rates different than thenative resolutions. For example, the native video data can be at a firstvideo frame rate, such as 24 fps. But, the video transformations can beperformed on every other frame and synthetic images can be output at aframe rate of 12 fps. Alternatively, the transformed video data can beinterpolated from the 12 fps rate to 24 fps rate by interpolatingbetween two of the transformed video frames.

In another example, prior to performing the video transformations, theresolution of the native video data can be reduced. For example, whenthe native resolution is 512 by 512 pixels, it can be interpolated to a256 by 256 pixel array using a method such as pixel averaging and thenthe transformation can be applied to the 256 by 256 array. Thetransformed video data can output and/or stored at the lower 256 by 256resolution. Alternatively, the transformed video data, such as with a256 by 256 resolution, can be interpolated to a higher resolution, suchas its native resolution of 512 by 512, prior to output to the displayand/or storage. The coarsening of the native video data prior toapplying the video transformation can be used alone or in conjunctionwith a coarser frame rate.

As mentioned above, the native video data can also have a color depth.The color depth can also be coarsened prior to applying thetransformations to the video data. For example, the color depth might bereduced from 40 bits to 24 bits prior to applying the transformation.

As described above, native video data from a live video can be augmentedwith virtual data to create synthetic images and then output inreal-time. In particular embodiments, real-time can be associated with acertain amount of latency, i.e., the time between when the native videodata is captured and the time when the synthetic images includingportions of the native video data and virtual data are output. Inparticular, the latency can be less than 100 milliseconds. In otherembodiments, the latency can be less than 50 milliseconds. In otherembodiments, the latency can be less than 30 milliseconds. In yet otherembodiments, the latency can be less than 20 milliseconds. In yet otherembodiments, the latency can be less than 10 milliseconds.

The interface 511 may include separate input and output interfaces, ormay be a unified interface supporting both operations. Examples of inputand output interfaces can include displays, audio devices, cameras,touch screens, buttons and microphones. When acting under the control ofappropriate software or firmware, the processor 501 is responsible forsuch tasks such as optimization. Various specially configured devicescan also be used in place of a processor 501 or in addition to processor501, such as graphical processor units (GPUs). The completeimplementation can also be done in custom hardware. The interface 511 istypically configured to send and receive data packets or data segmentsover a network via one or more communication interfaces, such aswireless or wired communication interfaces. Particular examples ofinterfaces the device supports include Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces, andthe like.

In addition, various very high-speed interfaces may be provided such asfast Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces,HSSI interfaces, POS interfaces, FDDI interfaces and the like.Generally, these interfaces may include ports appropriate forcommunication with the appropriate media. In some cases, they may alsoinclude an independent processor and, in some instances, volatile RAM.The independent processors may control such communications intensivetasks as packet switching, media control and management.

According to various embodiments, the system 500 uses memory 503 tostore data and program instructions and maintained a local side cache.The program instructions may control the operation of an operatingsystem and/or one or more applications, for example. The memory ormemories may also be configured to store received metadata and batchrequested metadata.

The system 500 can be integrated into a single device with a commonhousing. For example, system 500 can include a camera system, processingsystem, frame buffer, persistent memory, output interface, inputinterface and communication interface. In various embodiments, thesingle device can be a mobile device like a smart phone, an augmentedreality and wearable device like Google Glass™ or a virtual reality headset that includes a multiple cameras, like a Microsoft Hololens™. Inother embodiments, the system 500 can be partially integrated. Forexample, the camera system can be a remote camera system. As anotherexample, the display can be separate from the rest of the componentslike on a desktop PC.

In the case of a wearable system, like a head-mounted display, asdescribed above, a virtual guide can be provided to help a user record aMVIDMR. In addition, a virtual guide can be provided to help teach auser how to view a MVIDMR in the wearable system. For example, thevirtual guide can be provided in synthetic images output to head mounteddisplay which indicate that the MVIDMR can be viewed from differentangles in response to the user moving some manner in physical space,such as walking around the projected image. As another example, thevirtual guide can be used to indicate a head motion of the user canallow for different viewing functions. In yet another example, a virtualguide might indicate a path that a hand could travel in front of thedisplay to instantiate different viewing functions.

FIG. 6 shows a top-down diagram of a damage detection portal 602arranged as a gate, configured in accordance with one or moreembodiments. FIG. 7 shows a perspective diagram of the same damageportal 602. The damage detection portal 602 is configured as a gatethrough which a vehicle may be driven. The damage detection portal 602includes a number of cameras, such as the cameras 604, 606, 608, and610. The camera 604 is configured to point toward the front as thevehicle drives through the gate. The camera 606 is configured to pointtoward the back as the vehicle leaves the gate. The camera 610 isconfigured to point toward the interior area of the gate. The camera 608is configured to point down toward the top of the vehicle. The camera612 is configured to point up toward the undercarriage of the vehicle.Various configurations of cameras are possible.

In particular embodiments, an image of a vehicle, for instance an imageof the vehicle's undercarriage, may be created from two or more imagescaptured by one, two, or more cameras. For example, the vehicle may bedriven over two or more undercarriage cameras, which may each captureimages of a portion of the vehicle's undercarriage. Those images maythen be combined to yield a more complete image of the vehicle'sundercarriage, for example, by including portions of the undercarriagethat are not visible at the same time to a single camera.

In particular embodiments, an image of a vehicle may be created in aninteractive fashion. For example, by creating an image of a vehicle'sundercarriage based on different images captured with multiple cameras,a user may be able to change the view direction and look behind portionsof the undercarriage by switching to a camera with a different view. Asanother example, one or more cameras may be movable, for instance bybeing mounted on a track and/or gimbal. In this way, the system mayallow a camera to be repositioned to attain a different viewpoint, forinstance to look behind an object in the undercarriage.

In particular embodiments, two or more of the cameras associated withthe damage detection portal 602 may be synchronized. When cameras aresynchronized, they may be configured to capture images at the same timeor at nearly the same time. Alternatively, or additionally, synchronizedcameras may be configured to capture images that are staggered in timeby a fixed time period. By employing synchronized cameras, the imagescaptured from the cameras may be more easily linked. For instance,synchronizing cameras on the left and right side of the damage detectionportal may ensure that in a given image precisely the same portion ofthe vehicle is captured on the right side as by the corresponding cameraon the left side.

FIG. 8 shows a top-down diagram of a damage detection portal 802arranged as a tunnel, configured in accordance with one or moreembodiments. FIG. 9 shows a perspective diagram of the same damageportal 802.

In particular embodiments, a damage detection portal may be configuredas a turntable. In such a configuration, a vehicle may first bepositioned onto the turntable. The turntable may then rotate to presentthe vehicle at different angles to one or more fixed cameras.Alternatively, a turntable configuration may leave the vehicle in afixed position while a camera assembly rotates around the vehicle. Asyet another example, both the vehicle and the camera assembly may berotated, for instance in opposite directions.

According to various embodiments, in a turntable configuration, theturntable may rotate any suitable amount. For instance, the turntablemay rotate 360 degrees, 720 degrees, or 180 degrees.

FIG. 10A shows a top-down views of a damage detection portal 1002,configured in accordance with one or more embodiments. The damagedetection portal 1002 is configured to employ a drive path 1008 that iscurved both entering and leaving the damage detection portal 1002. Dueto the configuration of the cameras and the drive path, the camera 1004is configured to capture images of a vehicle head-on at a neutralelevation (also referred to as a “hero shot”) as the vehicle drivestoward the camera and then turns into the damage detection portal.Similarly, the camera 1006 is configured to capture images of thevehicle tail-on at a neutral elevation as the vehicle leaves the damagedetection portal 1002 and then curves away from the portal.

FIG. 10B shows a top-down views of a damage detection portal 1010,configured in accordance with one or more embodiments. The damagedetection portal 1010 is configured to employ a drive path 1014 that iscurved both entering and leaving the damage detection portal 1010. Dueto the configuration of the cameras and the drive path, the camera 1018is configured to capture images of a vehicle tail-on at a neutralelevation as the vehicle drives toward the camera and then turns intothe damage detection portal. Similarly, the camera 1012 is configured tocapture images of the vehicle head-on at a neutral elevation as thevehicle leaves the damage detection portal 1002 and then curves awayfrom the portal.

FIG. 11A shows a top-down views of a damage detection portal 1102,configured in accordance with one or more embodiments. The damagedetection portal 1102 is configured to employ a drive path 1108 that isstraight entering but curved leaving the damage detection portal 1102.Due to the configuration of the cameras and the drive path, the camera1108 is configured to capture images of a vehicle head-on at a neutralelevation as the vehicle leaves the damage detection portal 1102 andthen curves away from the portal. Similarly, the camera 1106 isconfigured to capture images of the vehicle tail-on at a neutralelevation as the vehicle leaves the damage detection portal 1102 andthen curves away from the portal.

FIG. 11B shows a top-down views of a damage detection portal 1110,configured in accordance with one or more embodiments. The damagedetection portal 1110 is configured to employ a drive path 1114 that iscurved entering and straight leaving the damage detection portal 1110.Due to the configuration of the cameras and the drive path, the camera1118 is configured to capture images of a vehicle head-on at a neutralelevation as the vehicle drives toward the camera and then turns intothe damage detection portal. Similarly, the camera 1114 is configured tocapture images of the vehicle tail-on at a neutral elevation as thevehicle turns into the damage detection portal.

FIG. 12 shows a side-view diagram of a damage detection portal 1202 as avehicle drives through it, configured in accordance with one or moreembodiments. The damage detection portal 1202 is positioned on a ramp1204. In this way, the camera 1208 can capture a frontal view of thevehicle head-on (i.e., a “hero shot”) as the vehicle drives up the rampbefore it levels off into the damage detection portal. Similarly, thecamera 1206 can capture a rear view of the vehicle tail-on as thevehicle leaves the damage detection portal and drives down the ramp.

FIG. 27 shows a side-view of a damage detection portal 2702 as a vehicle2708 drives through it, configured in accordance with one or moreembodiments. The damage detection portal 2702 may include some number ofcameras arranged as described with respect to FIGS. 6-12 or arranged ina different configuration. In addition, the damage detection portal 2702may be configured to communicate with the cameras 2704 and/or 2706,which may capture images of the vehicle 2708 before and/or after itenters the damage detection portal 2702.

According to various embodiments, the diagrams shown in FIGS. 6-12 and27 illustrate only a few of the possible configurations of a damagedetection portal. Various configurations are possible and in keepingwith the techniques and mechanisms described herein.

FIG. 13 illustrates the capture of image data via a damage detectionportal, configured in accordance with one or more embodiments. Accordingto various embodiments, one or more cameras may be configured to capturea whole-vehicle image such as the image 1302 as the vehicle drivesthrough the portal. Alternatively, or additionally, one or more camerasmay be configured to capture a closeup view such as the image 1304 asthe vehicle drives through the portal. By combining these views, a usermay be able to select a portion of a whole-vehicle image and then zoomin to a view captured by a closeup camera.

FIGS. 14-16 show simulated images generated from a damage detectionportal, configured in accordance with one or more embodiments. In FIG.14 , a vehicle is shown prior to entering into the portal. In FIG. 15 ,the vehicle is shown in the portal. In FIG. 16 , a vehicle is shownleaving the portal.

FIGS. 17-19 show images captured via a damage detection portal andpresented in a user interface, generated in accordance with one or moreembodiments. In FIG. 17 , images of a vehicle captured from differentperspectives are shown. When one of the images is selected, it may beenlarged, as shown in FIG. 18 . As shown in FIG. 19 , one or more imagesmay be captured of the undercarriage of the vehicle.

FIG. 20 illustrate a portion of the user interface in which detecteddamage is shown, configured in accordance with one or more embodiments.In FIG. 20 , detected damage is illustrated on a top-down view of thevehicle as a heatmap. On the right, a list of components of the vehicleis shown, along with a status bar and percentage indicating the degreeof coverage provided by the captured images.

FIGS. 21-25 illustrate images captured via the damage detection portaland presented in the user interface, configured in accordance with oneor more embodiments. In some embodiments, an image may be selected byclicking on or touching damage represented in the top-down view. Forinstance, clicking on or touching the damage shown on the left doorpanels in FIG. 20 may lead to the presentation of the image shown inFIG. 21 .

In some implementations, a selected image may be a portion of a closeupmulti-view interactive digital media representation (MVIDMR) of theselected region. The closeup MVIDMR may depict the selected portion ofthe vehicle from different perspectives. The user may navigate betweenthese different perspectives by, for example, clicking and dragging amouse, or touching and dragging on a touch screen. For example, in FIG.21 , the user has selected an area in the center of the image and thendragged to one side, leading the user interface to present the imageshown in FIG. 22 , which depicts the same area of the vehicle from adifferent perspective. A similar operation is shown in FIGS. 23 and 24 ,which depict a different closeup MVIDMR of the back left area of thevehicle.

According to various embodiments, damage to the vehicle may beidentified in a list, such as that shown in FIG. 25 . The identifieddamage may include information such as the location that was damaged,the type of damage (e.g., a dent, or paint damage), a confidence levelassociated with the detected damage, and/or the severity of the damage.

FIG. 26 illustrates an overview method 2600 for the operation of adamage detection portal, configured in accordance with one or moreembodiments. According to various embodiments, the method 2600 may beperformed at a damage detection portal or computing device incommunication with a damage detection portal. Alternately, oradditionally, some or all of the method 2600 may be performed at aremote computing device such as a server. The method 2600 may be used todetect damage to any of various types of objects. However, for thepurpose of illustration, many examples discussed herein will bedescribed with reference to vehicles.

A request to perform damage detection is received at 2602. According tovarious embodiments, the request may be based on user input. Forinstance, a user may transmit a request to initiate damage detection.Alternatively, or additionally, the request may be automaticallygenerated. For instance, damage detection may begin automatically whenthe system is activated.

Data from one or more sensors is collected at 2604. According to variousembodiments, the sensor data may include information collected from oneor more pressure sensors, cameras, light sensors, or any other suitablesensors.

A determination is made at 2606 as to whether an object is detected. Insome implementations, the sensor data may be used to determine when anobject is approaching the damage detection portal. The determination maybe limited, for instance detecting whether a laser sensor has beeninterrupted or a pressure panel has been tripped. Alternatively, thedetermination may involve performing sophisticated object recognitionbased on visual data collected from one or more cameras.

When an object is detected, then at 2608 image data from one or moredamage detection portal cameras is collected. As discussed herein, adamage detection portal may have multiple cameras that capture imagedata of the object at different angles and from different viewpoints.

Object travel speed is determined at 2610. In some implementations, theobject travel speed may be determined based on one or more sensors suchas cameras, pressure sensors, laser sensers, radar sensors, sonarsensors, or any other suitable sensors. The object travel speed may beused to inform the rate at which visual data is captured. For instance,visual data capture may be adjusted so as to capture a relativelyconstant amount of visual data regardless of object speed. When avehicle is traveling faster, for example, cameras may be configured tocapture images at a more rapid pace than when a vehicle is travelingmore slowly.

A determination is made at 2612 as to whether the object has departedfrom the damage detection portal. According to various embodiments, thedetermination may be made based on one or more of a combination of datasources. For example, a pressure sensor may detect when an object hasmoved away from the portal. As another example, image information may beused to determine that an object is no longer present in the area of theportal. As yet another example, a laser or other sensor may be detectwhen an object has passed a designated point along a path.

When the object has departed from the damage detection platform, anMVIDMR of the object is constructed at 2614. According to variousembodiments, image data may be used to construct an overall MVIDMR ofthe entire object. Additionally, one or more focused MVIDMRs may beconstructed of particular areas or components of the object. Forexample, a focused MVIDMR of a vehicle component may be constructed. Asanother example, a focused MVIDMR of a portion of a vehicle in whichdamage has been detected may be constructed.

Damage to the object based on the MVIDMR is identify at 2616. Accordingto various embodiments, any of a variety of techniques may be used toperform damage detection. Examples of such damage detection techniquesare described throughout the application, for instance with respect tothe FIGS. 1-5 .

Information is stored on a storage device at 2618. According to variousembodiments, storing the information may involve transmittinginformation via a communication interface over a network to a remotestorage location and/or storing the information on a local storagedevice. The information stored may include, but is not limited to: rawimage and/or video data, sound data captured as the object passedthrough the portal, one or more MVIDMRs constructed as discussed atoperation 2616, and/or damage detection information determined asdiscussed at operation 2616.

According to various embodiments, although the configuration of camerasis referred to herein as a damage detection portal, the configuration ofcameras may be used for other purposes, such as to record a video of thevehicle that includes multiple perspectives.

According to various embodiments, although the object captured by thedamage detection portal is referred to herein as a vehicle, informationabout other types of objects may be captured in a similar fashion. Forexample, a damage detection portal may be used to capture informationabout a patient in a medical setting. As another example, a damagedetection portal may be used to capture information about an individualfor security purposes. As yet another example, a damage detection portalmay be used to capture information about animals. As still anotherexample, a damage detection portal may be used to capture informationabout objects on an assembly line. A variety of configurations andapplications are possible.

FIG. 28 illustrates an image extraction method 2800, performed inaccordance with one or more embodiments. The method 2800 may be used toselect one or more standardized images from a set of images.

In some implementations, a standard image may be an image of an objectthat is captured from a relatively standardized position relative to anobject. For example, one standard image of a vehicle may be captured bya camera located directly in front of and slightly above the vehicle. Asanother example, another standard image of the vehicle may be capturedby a camera located at a 30-degree angle from the front right of thevehicle. As discussed herein, various types of criteria may be used tospecify a standardized image.

According to various embodiments, standard images may be used in avariety of contexts. For example, automotive wholesale and retailoperations often employ a set of standard images to present cars topotential buyers. When using conventional techniques, obtaining theseimages in a consistent and efficient way is often challenging andrequires a significant amount of training for human camera operators.

At 2802, a request to capture a standardized image of an object isreceived. In some implementations, the request may be generated based onuser input. Alternatively, the request may be generated as part of anautomatic process.

One or more criteria for selecting a standardized image are identifiedat 2804. According to various embodiments, a variety of selectioncriteria may be used. A few of the possible selection criteria aredescribed in the following paragraphs.

In a first example, a selection criteria may be related to the portionof an image frame occupied by an object. For instance, an image of anobject in which the object occupies a greater percentage of the imageframe may be designated as a superior standardized image.

In a second example, a selection criteria may be related to a locationwithin the image frame occupied by the object. For instance, an image ofan object in which the object occupies a central position within theimage frame, and/or a portion of the image frame with the best focus,may be designated as a superior standardized image.

In a third example, a selection criteria may be related to a distancebetween the object and the camera. For instance, an image of an objectmay be designated as a superior standardized image when the distancebetween the object and the camera is closer to a designated distance.

In a fourth example, a selection criteria may be related to objectorientation. For instance, an image of an object may be designated as asuperior standardized image when the object is located more closely witha designated orientation with respect to the camera.

In a fifth example, a selection criteria may be related to featurecoverage. For instance, an image of an object may be designated as asuperior standardized image when one or more designated features of theobject are captured more completely in the image frame.

In particular embodiments, more than one selection criteria may be used.In such a configuration, one or more configuration parameters may beused to specify how to balance between the various selection criteria.For example, a standardized image may be identified as (1) beingcaptured from within a particular range of angular orientations withrespect to the object, (2) being captured from within a particular rangeof distances from the object, (3) including all of the object within theimage frame, and (4) maximizing the portion of the image frame occupiedby the object.

At 2806, an image in an image set is identified for analysis. Accordingto various embodiments, image sets may be captured in any of variousways. For example, a fixed camera setup may be used, where the object(e.g., a vehicle) is moved past one or more cameras. As another example,the object may remain fixed, while one or more cameras are moved aroundthe object. For instance, a person may capture images of the objectusing a hand-held camera or mobile computing device such as a mobilephone.

In particular embodiments, images may be analyzed from the image set inany suitable order. For example, images may be analyzed in sequence, inparallel, at random, or in any suitable order. As another example,images may be pre-processed to identify candidate images that are morelikely to meet one or more selection criteria.

At 2808, a determination is made as to whether the identified imagemeets one or more threshold criteria. According to various embodiments,as discussed above, one or more selection criteria may be used. Forinstance, an image may only be suitable as a standardized image if it isidentified as (1) being captured from within a particular range ofangular orientations with respect to the object, (2) being captured fromwithin a particular range of distances from the object, and (3)including all of the object within the image frame.

At 2812, a determination is made as to whether the identified image is abetter match for the selection criteria than any previously selectedimage in the set. When the image is so identified, then at 2812 it isdesignated as the currently selected standardized image. For example, ofthe images identified at 2808 as meeting the threshold selectioncriteria, the image in which the object occupies a greater proportion ofthe image frame may be identified as the superior image.

At 2814, a determination is made as to whether to identify an additionalimage in the image set for analysis. In some implementations, each imagein the image set may be analyzed. Alternatively, or additionally,successive images may be analyzed until one is identified thatsufficiently meets the criteria for a standardized image.

At 2816, one or more processing operations are optionally applied to thecurrently selected image. In some implementations, one or more of avariety of processing operations may be employed. For example, an imagemay be improved, for instance by blurring the background around theobject. As another example, the object may be segmented out from theframe and background, and the background may be blurred or replaced witha custom background for imaging purposes.

At 2818, the currently selected image is stored as the standardizedimage. According to various embodiments, storing the image may involvetransmitting the image to a local or remote storage device.

FIG. 29 describes a method 2900 for calibrating a damage detectionportal, performed in accordance with one or more embodiments. The method2900 may be performed at a computing device in communication with adamage detection portal such as the portals discussed throughout theapplication as filed.

A request to calibrate a damage detection portal is received at 2902. Insome implementations, the request may be generated based on user input.Alternatively, or additionally, the request may be generatedautomatically. For instance, the request may be generated when thedamage detection portal is first activated, when the position of acamera changes, or at periodic intervals.

A camera is selected for analysis at 2904. As discussed herein, a damagedetection portal may be associated with multiple cameras configured infixed positions. According to various embodiments, cameras may beselected for analysis in sequence, at random, in parallel, or in anysuitable order.

Images of an object moving through the damage detection portal capturedby the selected camera are identified at 2906. In some implementations,the images may be of a three-dimensional object such as a vehicle.Alternately, a two-dimensional object such as an flat surface may beused.

In some implementations, the object moving through the damage detectionportal may be an object for which the portal is configured to detectdamage. Alternatively, a calibration object such as a checkerboardprinted on a flat surface may be used.

If a three-dimensional object is used, then at 2908 a three-dimensionalmodel of the object is determined based on the identified images. Then,at 2910, timing information identifying when the images were capturedmay be used to determine a position of the 3D model with respect to thecamera over time. At 2912, one or more correspondence points for the 3Dmodel at designated points in time may be determined. Eachcorrespondence point may correspond to a particular identifiable portionof the object, such as a joint in a skeleton frame of the object.

In the event that a two-dimensional calibration print such as acheckerboard is used for calibration, then operations 2908-2912 mayinvolve determining a position of the camera relative to one or morepoints on the calibration print at one or more designated points intime.

A determination is made at 2914 as to whether to select an additionalcamera for analysis. In some implementations, each camera in the damagedetection portal may be analyzed until all cameras have been calibrated.

An alignment for the correspondence points is determined at 2916. Forexample, the alignment may be determined by identifying correspondencepoints that are known to two or more cameras at the same time. Forinstance, different three-dimensional models may be constructed based onimages captured from two different cameras. However, the differentthree-dimensional models may share correspondence points such as thesame joint in a three-dimensional skeleton of the object. Because thetime at which each image is captured is recorded, a determination can bemade that a given alignment point has a first position with respect toone camera and a second position with respect to a second camera. Suchinformation may be determined for multiple alignment points and multiplecameras.

An alignment transform is then determined at 2918 for each camera basedon the alignment. The alignment transform may identify the relativeposition and direction of the camera with respect to the alignmentpoints at a designated point in time. The alignment transforms arestored as camera pose calibration information at 2920. That is, therelative position of the cameras with respect to known correspondencepoints provides the alignment information that may be subsequently usedto process and aggregate visual data collected by the cameras.

FIG. 30 illustrates a method 3000 for damage detection, performed inaccordance with one or more embodiments. According to variousembodiments, the method 3000 may be performed at a computing device incommunication with a damage detection portal.

A request to detect damage to an object is detected at 3002. In someimplementations, the request may be generated based on user input.Alternatively, or additionally, the request may be generatedautomatically. For instance, the request may be generated when thedamage detection portal detects that an object has entered or ispredicted to enter the damage detection portal. Such a determination maybe made based on input from one or more cameras, pressure sensors, lasersensors, or other such sensors.

Image data of the object is collected at 3004. In some implementations,the image data may include sets of images captured by different camerasassociated with the damage detection portal. Each of the images may beassociated with a synchronized time stamp. In particular embodiments,the cameras may be synchronized to capture images at the same time sothat the object may be captured simultaneously from differentperspectives.

Motion of the object is estimated at 3006. According to variousembodiments, motion may be estimated for images captured by one or moreof the cameras. Then, motion estimation information may be propagated toother cameras associated with the damage detection portal. The motioninformation may be determined by estimating an object model associatedwith the object. Then, a location of the object model may be determinedat successive points in time.

The image data is mapped to an object model at 3008. In someimplementations, image data may be mapped if the damage detection portalhas been calibrated. In such a configuration, a semantic mesh may begenerated according to the motion estimation. The mesh may then bemapped to each frame based on the extrinsic camera calibrationinformation. For instance, after the motion of an object such as avehicle has been estimated, then the position of the object model withrespect to each camera at a designated point in time may be determinedbased on the known relation of the cameras to one another. Then, damagemay be mapped to the object model (e.g., a two-dimensional top-downview) through the semantic mesh.

FIGS. 32-35 illustrate examples of a semantic mesh, generated inaccordance with one or more embodiments. The semantic mesh is generatedbased on multiple perspective view images. Because each image is mappedto a top-down view, the semantic mesh can be mapped back to theperspective view images.

FIGS. 36-41 illustrate examples of a component analysis applied to thesemantic mesh, performed in accordance with one or more embodiments. InFIGS. 36-41 , different components of the vehicle, such as door panels,wheels, windshields, and windows, are shown in different colors.

According to various embodiments, image data may be mapped if the damagedetection portal is uncalibrated. In such a configuration, a proceduresuch as a neural network may be run for each frame to map the pixels ofthe frame to a standardized object model such as a two-dimensionaltop-down view of an object.

Capture coverage information is determined at 3010. In someimplementations, the capture coverage information may indicate portionsof the vehicle that have been captured, or not, in the detected imagedata. For example, the capture coverage information may indicate whichpanels of a vehicle are covered in the image data, and provide anindication of the degree of coverage, such as a percentage. Suchinformation may be determined by reasoning about the object model (e.g.,a two-dimensional top-down image or a three-dimensional skeleton modelof the object).

Damage to the object is detected at 3012. Techniques for detectingdamage are discussed throughout the application. At 3014, adetermination is made as to whether damage is detected. If damage isdetected, then the detected damage is mapped to a standard object viewat 3016.

In some implementations, damage detection may involve aggregatinginformation from various cameras. For example, multiple cameras may eachcapture multiple views of a vehicle door from multiple viewpoints as thevehicle passes through the damage detection portal. When the damagedetection portal has been calibrated, these different viewpoints may becorrelated so that the same portion of the vehicle is captured fromdifferent known perspectives at the same point in time. This informationmay then be used to provide an aggregated and/or filtered damageestimate, for instance in a standard top-down view of the object.

In particular embodiments, damage detection information may be used toreason about the damage in a standard object model such as atwo-dimensional top-down view or a three-dimensional skeleton. Forexample, the aggregated information may be used to determine that a dentto a vehicle covers half of the right front door in the object model, sothe aggregated damage information may include information indicatingthat 50% of the right front door has been damaged.

One or more localized multi-view interactive digital mediarepresentations (MVIDMRs) at 3018. In some implementations, an MVIDMR ofa portion of the object identified as damaged may be automaticallygenerated when damage is detected. Alternatively, or additionally, anMVIDMR may be generated of a designated portion of the object regardlessof whether damage has been detected. Additional techniques relating tolocalized MVIDMR generation are discussed with respect to the method3100 shown in FIG. 31 .

The damage detection information is stored at 3020. In someimplementations, storing the damage detection information may involveaccessing a local storage device. Alternatively, or additionally, thedamage detection information may be transmitted to a remote device via anetwork.

FIG. 31 illustrates a method 3100 for localized MVIDMR generation,performed in accordance with one or more embodiments. The method 3100may be performed at any suitable computing device. For instance, themethod 3100 may be performed at a computing device in communication witha mobile phone or a damage detection portal.

A request to generate a localized MVIDMR of an object portion isreceived at 3102. In some implementations, the request may be generatedbased on user input. Alternatively, or additionally, a request may begenerated automatically. For example, the method 3100 may be performedto generate a localized MVIDMR of an area of an object identified asbeing damaged. For instance, the method 3000 discussed with respect toFIG. 30 may result in the identification of one or more components of anobject that have been damaged. Then, a localized MVIDMR may beautomatically generated.

According to various embodiments, the object portion may be identifiedin any of various ways. For instance, the object portion may beidentified as an area or component on an object model associated withthe object. The object model could be a two-dimensional top-down view, athree-dimensional skeleton, or any suitable object model.

A set of images that includes the object portion is selected at 3104based on a mapping from the object model. In some implementations, theimages may be collected as the object passes through a damage detectionportal. Alternatively, or additionally, one or more images may becaptured by a handheld device such as a mobile phone. The images may bemapped to the object model. Then, the correspondence between the imagesand the object model may be used to determine which images include theidentified portion of the object.

One or more of the selected images may be optionally cropped around theobject portion at 3106. In some implementations, cropping an image mayinvolve first identifying a portion of the image that corresponds to theidentified object component. The identification may involve performingobject identification on the identified image. Alternatively, oradditionally, the object component mapping may be used to identify theportion of the image that maps to the identified portion of the object.

In particular embodiments, an image may be cropped so that the image iscentered around the object portion of interest. Alternatively, oradditionally, an image may be cropped so that the object portion ofinterest occupies the majority of the frame.

One or more synthetic images of the object portion may be optionallygenerated at 3108. In some implementations, synthetic image generativemay involve a light field technique. For instance, each of a pluralityof two-dimensional images may be elevated into a respective multi-planeimages. The multiplane images around the synthetic viewpoint may becombined to generate a synthetic multiplane image at the syntheticviewpoint. The synthetic multiplane image may then be projected backinto a two-dimensional image. A synthetic image may provide for a moreseamless transition between frames.

A localized multi-view interactive digital media representation (MVIDMR)is generated at 3110. The localized MVIDMR is then stored at 3112, whichmay involve storing the localized MVIDMR on a local storage deviceand/or transmitting the localized MVIDMR to a remote location. Accordingto various embodiments, the MVIDMR may be generated in accordance withtechniques and mechanisms throughout the application as filed.

FIG. 42 illustrates an overview method 4200, performed in accordancewith one or more embodiments. The method 4200 may be performed at acomputing device in communication with an undercarriage imaging system.The undercarriage imaging system may be integrated with a damagedetection portal.

Images of a vehicle undercarriage are captured at 4202. According tovarious embodiments, the images may be captured as the vehicle movesover one or more cameras that are arranged beneath the vehicle. Forexample, the vehicle may be driven over the cameras, or may be movedover the cameras via a belt or other such system. Examples of anundercarriage imaging system are discussed in additional detail withrespect to FIG. 43 , and throughout the application.

In particular embodiments, the undercarriage imaging system may operateas part of, or in communication with, a vehicle imaging system that maybe configured as a gate, tunnel, or gantry. For example, the vehicleimaging system may read a vehicle identifier such as a vehicleidentification number (VIN), a license plate number, another type ofplacard having an identification number and arranged on or in thevehicle (e.g., in the vehicle's windshield area), an RFID tag, or anyother suitable mechanism for identifying the vehicle. As anotherexample, the vehicle imaging system may be used to capture vehicleposition information, which may be synchronized with undercarriageimaging data captured by the undercarriage imaging system.

In particular embodiments, the undercarriage imaging system maycommunicate with a communication device such as a tablet, mobile phone,laptop computer, or other such system. For instance, a user may enter avehicle's identification number on an application running on a mobilephone, which may then communicate with the undercarriage imaging systemto capture image data of the vehicle's undercarriage.

Some or all of the captured images are stitched together at 4204. Insome implementations, the images may be stitched together in any ofvarious ways. For example, feature points may be identified insuccessive images and then matched to one another. Then, successiveimages may be stitched together based on the matching feature points. Asanother example, images may be used to construct a three-dimensionalmodel such as a point cloud of the vehicle undercarriage. Then, thethree-dimensional model may be used to stitch together different images.Examples of techniques for stitching together captured images arediscussed in additional detail with respect to the methods 4800 and 4900shown in FIGS. 48 and 49 .

One or more MVIDMRs are constructed based on the images at 4206.According to various embodiments, an MVIDMR may be constructed based onimages captured from a single camera. Alternatively, images capturedfrom multiple cameras may be used to construct the MVIDMR. Techniquesfor constructing MVIDMRs are discussed throughout the application.

According to various embodiments, any of a variety of cameraconfigurations may be possible. In some embodiments, a single camera maybe used. Such a camera may be positioned in an upward facing direction.The camera may be equipped with a fisheye lens. Alternatively, oradditionally, the camera may be a 360-degree camera, a 180-degreecamera, an ultrawide camera, a conventional camera, line-scan camera,conventional camera configured to operate as a line-scan camera, and/orany other suitable image or data capture device.

In some embodiments, multiple cameras may be configured on the ground ina line across which a vehicle travels. Alternatively, multiple camerasmay be configured on the ground in a different pattern. For instance, a“+” pattern or a grid pattern may be used. Including cameras in aconfiguration in which there is variation in two dimensions may help toimprove image stitching, particularly for vehicles with relativelyuniform undercarriages and few feature points. Hardware and/or softwaremay be used to trigger synchronized camera capture.

In some embodiments, a mirror strip may be used in conjunction with asingle camera. In such a configuration, the mirror may be angled so thatthe undercarriage is imaged in the mirror.

In some embodiments, a mirror strip may be used in conjunction withmultiple cameras. In such a configuration, the mirror may be angled sothat the undercarriage is imaged in the mirror. Using multiple mirrorsmay provide for multiple perspective viewpoints of the same portion ofthe vehicle undercarriage, which may provide for various MVIDMRconstruction possibilities.

In some embodiments, a mirror and camera configuration may be used inconjunction with one or more additional mirrors not configured to employthe mirror. For example, one or more mirrors may be placed behind orbeside the mirror. The additional cameras may be used to performoperations such as stitching together images of the sides of the vehiclenot easily visible in the mirror. Alternatively, or additionally, theadditional cameras may be used to help with feature point tracking andmatching, for instance if the images from the mirror provideinsufficient feature points.

In some embodiments, one or more cameras may be used in conjunction witha ramp. For example, multiple cameras may be positioned on the groundpointing upwards as the vehicle travels up and over the ramp.

In some embodiments, one or more line sensors may be used. A line sensoris similar to a scanner technology in which image data is captured by animage sensor in a line configuration.

In some embodiments, one or more laser scanners, time-of-flight sensors,and/or structured light sensors may be used. According to variousembodiments, such sensors may help to model the undercarriage in threedimensions, and may be used instead of or in conjunction with one ormore traditional cameras.

FIG. 43 illustrates an undercarriage imaging system 4300, configured inaccordance with one or more embodiments. The undercarriage imagingsystem 4300 may be integrated into a damage detection portal, such asthe portals shown in FIGS. 8, 9, and 13 . The undercarriage imagingsystem 4300 includes a vehicle 4302 that travels over a camera 4306 anda mirror 4304.

According to various embodiments, the mirror 4304 may be a rectangularstrip having a vertical height lower than the undercarriage. Variousconfigurations may be used. For example, the mirror may be configured tobe more narrow than the width of the vehicle wheels so that the vehiclemay drive over it. As another example, the mirror may be configured sothat it depresses flat if and when the vehicle's wheels pass over it,and then springs back into position after the vehicle's wheel or wheelshave moved off of it. As yet another example, the mirror may beconfigured to include multiple depressible panels so that a portion ofthe mirror over which a vehicle's wheel passes is depressed while adifferent portion of the mirror over which the vehicle's wheel does notpass remains in the angular position.

According to various embodiments, one or more cameras may be configuredin a fixed or variable orientation. For instance, a fixed orientation of45 degrees may be used. In this way, the camera 4306 may be configuredto capture a relatively vertical image of the undercarriage of thevehicle.

In some implementations, one or more cameras and/or one or more lightsmay be configured on a convex or concave surface or orientation. Forexample, convexity or concavity may allow visual angles that would notbe accessible on a flat surface or orientation.

In some implementations, the camera 4306 may capture one or more imagesor image portions of the undercarriage of the vehicle from a relativelyvertical angle by reflection on the mirror 4304. Alternatively, oradditionally, the camera 4306 may capture one or more images or imageportions of the undercarriage of the vehicle from a relativelyhorizontal angle. For instance, the camera 4306 may directly capture oneor more images of the undercarriage of the vehicle 4302 from an obliqueor relatively horizontal angle as the vehicle travels over the camera.

FIG. 44 illustrates an image captured by a single camera in a mirrorconfiguration, generated in accordance with one or more embodiments.Such an image is effectively split into different portions given thefield of view captured by the camera. The image portion A 4402corresponds to the portion of the undercarriage visible in the camera'sfield of view above the mirror. The top of the mirror is visible at4404. The image portion B 4406 is the portion of the image that iscaptured by reflection in the mirror, and illustrates some of the samefeatures shown in the image portion A but from a more vertical angle.The image portion C 4408 corresponds to the ground.

According to various embodiments, as part of the image stitchingprocess, an image such as that shown in FIG. 44 may be cropped toeliminate the image portion C 4408 and to separate the image portion A4402 from the image portion B 4406. Then, one or both of the separatedimages may be used to form a stitched image.

FIGS. 45A and 45B illustrate images generated by an undercarriageimaging system configured in accordance with one or more embodiments.FIG. 45A illustrates a single image of the vehicle undercarriagecaptured by a camera having a low focal length, which is commonlyreferred to as a fisheye lens. FIG. 45B illustrates the same image afterbeing transformed from a fisheye perspective to a flat projectionperspective.

FIGS. 46A, 46B, and 46C illustrate stitched images captured of theundercarriage of a vehicle by undercarriage capture systems configuredin accordance with one or more embodiments. FIG. 46A illustrates anundercarriage image stitched from individual images captured in a singlecamera and mirror configuration. FIG. 46B illustrates an undercarriageimage stitched from individual images captured in a single fisheyecamera configuration. FIG. 46C illustrates an undercarriage imagestitched from individual images captured in a wide-angle cameraconfiguration.

FIGS. 47A and 47B illustrate images generated by an undercarriageimaging system configured in accordance with one or more embodiments.FIG. 47A illustrates a single image of a slide of the vehicleundercarriage captured by a 360-degree camera. FIG. 47B illustrates astitched image of the entire vehicle undercarriage constructed bystitching together a sequence of images captured by the 360-degreecamera.

According to various embodiments, an undercarriage capture system mayinclude various types and combinations of sensors. In a first example,two or more cameras may be directed at a mirror, as discussed withrespect to FIG. 43 . Such cameras may be arranged in parallel. Anexample of such a configuration is shown in FIG. 55A, which the cameras5502, 5504, and 5506 are arranged in parallel and directed at the mirror5508. Alternatively, or additionally, two or more cameras may bearranged at different angles, for instance to capture the same portionof the undercarriage from different perspectives. An example of such aconfiguration is shown in FIG. 55A, which the cameras 5512, 5514, and5516 are arranged at different angles and directed at the mirror 5518.

In a second example, one or more camera equipped with a fish-eye lensmay be used to detect vehicle motion, while one or more line cameras maybe used to capture data for constructing a stitched image.

In a third example, one or more cameras may be configured to capturethermal images. Then, image data from the thermal image may be presentedtogether with or separate from visual-spectrum image data. For instance,thermal image data may be presented as an overlay (e.g., a heatmap) ontop of visual-spectrum image data.

In a fourth example, one or more cameras may be arranged in a portableconfiguration. For instance, one or more cameras may be arranged on afold-out panel that can be arranged on a roadway.

In a fifth example, one or more lighting sources such as LED-drivenpanels may be employed. Such an arrange may provide for lower exposuretimes for image acquisition.

According to various embodiments, any or all of the exampleconfigurations described herein may be employed in various arrangementsor combinations.

FIG. 48 illustrates a method A 4800 for stitching an undercarriageimage, performed in accordance with one or more embodiments. The method4800 may be performed at a computing device in communication with anundercarriage imaging system.

A request to construct a stitched image of a vehicle undercarriage isreceived at 4802. According to various embodiments, the request may begenerated manually based on user input. Alternatively, the request maybe generated automatically, for instance after a vehicle traverses anundercarriage imaging system.

A set of images of the vehicle undercarriage is identified at 4804.According to various embodiments, the set of images may be captured byone or more cameras configured as discussed herein. The images may becaptured as the vehicle passes over the one or more cameras.

In particular embodiments, two or more of the images may be captured ina time-synchronized fashion. For instance, different cameras arranged ina known configuration may be configured to capture images at specifictimes. For example, images may be captured by multiple cameras at thesame time. As another example, images may be captured by multiplecameras at a designated temporal offset.

One or more image transformations are optionally performed at 4806.According to various embodiments, any of a variety of suitable imagetransformations may be performed. For example, in the case of a fisheyelens, the image may be transformed via a fisheye projection from a widepanoramic or hemispherical image to a rectilinear image.

Feature points in the set of images are detected at 4808. According tovarious embodiments, feature points may be detected by analyzing eachimage to identify recognizable points. For example, feature points mayinclude lines, edges, points, corners, or other geometric figures. Asanother example, feature points may include words, color regions, orother visual elements. As yet another example, feature points mayinclude pipes, cables, structures, or other recognizable physicalcomponents.

Images are matched based on the detected feature points at 4810. In someimplementations, image matching may include linking feature points indifferent images. For instance, a word, edge, corner, or cable in oneimage may be linked to the corresponding word, edge, corner, or cable inanother image. The linkable of multiple feature points across two imagesmay provide for the creation of a transformation for stitching togetherthe two images.

An optimization problem using the detected feature points as crossconstraints is determined at 4812. A stitched image is determined bysolving the optimization problem at 4814. According to variousembodiments, the solution to the optimization problem may include one ormore transformations for linking images. For instance, images may betranslated, expanded, shrunk, sheared, or otherwise transformed so thatthe feature points of the different images align.

As an example, FIG. 53 shows the determination of a correspondencebetween images captured by the undercarriage system in accordance withone or more embodiments. In FIG. 53 , the images 5302 and 5304 arecaptured by a first camera, while the images 5306 and 5308 are capturedby a second camera, as a vehicle moves over the cameras. Colored linesindicate correspondences between tracking points in the image. Theaverage of the correspondences may allow the creation of atransformation for linking the images. For instance, a transformationmay be determined based on factors such as confidence in the detectedfeatures and matches and/or a cost function associated with one or moreparameters such as image differences and/or reprojection errors.

The stitched image is stored at 4816. According to various embodiments,stitched images may be stored on a local storage device. Alternatively,or additionally, stitched images may be transmitted to a remote machinevia a network.

In particular embodiments, one or more operations shown in FIG. 48 maybe omitted. For instance, instead of detecting feature points andmatching images at operation 4808, a transformation may be determinedbetween neighboring images by analyzing the absolute difference betweenthe successive images.

FIG. 49 illustrates a method B 4900 for stitching an undercarriageimage, performed in accordance with one or more embodiments. The method4900 may be performed at a computing device in communication with anundercarriage imaging system.

A request to construct a stitched image of the vehicle undercarriage isreceived at 4902. A set of images of the vehicle undercarriage isidentified at 4904. One or more image transformations are performed at4906. According to various embodiments, the operations 4902-4906 may besubstantially similar to the operations 4802-4806 discussed with respectto the method 4800 shown in FIG. 48 .

One or more camera poses are determined at 4908. In someimplementations, the pose of a camera may include information such asthe distance from the camera to the undercarriage and the angularposition of the camera with respect to the undercarriage.

In some embodiments, a camera pose may be determined in two dimensions.For example, when a camera is positioned at a fixed location andcalibrated, the pose of the camera relative to the vehicle may need tobe determined only in x and y dimensions, since the camera's orientationmay be known. A two-dimensional camera pose may be supplemented with adistance measure z between the camera and the vehicle to provide athree-dimensional camera pose.

In some embodiments, a camera pose may be determined in up to sixdimensions. For example, the camera's location relative to the vehiclemay be specified in three dimensions, while the camera's orientation maybe specified in another three dimensions (e.g., roll, pitch, and yaw).Such an arrangement may be particularly useful in, for instance, amobile configuration in which camera calibration may be difficult.

According to various embodiments, a camera pose may be determined viaany of various techniques. For example, one or more cameras may becalibrated so that a pose is known in advance. As another example, thepose for a camera may be determined by analyzing image data to determinea position and angle of the camera with respect to the vehicleundercarriage. As yet another example, calibration data may be used incombination with image data to determine a pose for a camera.

In particular embodiments, the angular position of a camera may be knownbased on calibration data. The distance from the camera to theundercarriage may then be determined. For instance, the distance may bedetermined by analyzing image data and/or data from a distance sensorsuch as a time-of-flight sensor.

A point cloud of the undercarriage is determined at 4910. In someimplementations, a point cloud may be determined by identifyingundercarriage features in an image, as discussed with respect to theidentification of feature points in operation 4808. Then, the featuresmay be positioned in three-dimensional space. For example, depth sensorinformation may be used to determine an approximate distance from thecamera for each feature. As another example, image data may be analyzedto estimate depth information.

In particular embodiments, one or more of various types of depth sensorsmay be used. For instance, a depth sensor may be implemented as atime-of-flight and/or structured light sensor.

According to various embodiments, a point cloud may be determinedseparately for individual images or groups of images. Then, thedifferent point clouds may be merged, for instance by linking featuresidentified in each image, as discussed with respect to the operations4810-4814 in FIG. 48 .

A virtual camera is created at 4912, and the point cloud is projectedonto the virtual camera. The virtual camera may be represented by apoint in space from which the image is taken. Alternatively, the virtualcamera may be represented by a virtual image plane. For instance, thepoint cloud may be projected onto a two-dimensional plane, which may beused as a canvas for creating the image.

An image transform that best fits the projection is determined at 4914.A stitched image is determined at 4916 based on the identifiedtransform. According to various embodiments, the transform may beselected so as to match image features corresponding to points in thepoint cloud as positioned in accordance with the projection determinedat 4912.

According to various embodiments, information captured by sensors isfrequently referred to herein as images or image data. However, itshould be noted that a wide variety of information may be captured bysensors. For instance, the captured information may include distance ordepth information captured by LIDAR or other types of sensors. When suchinformation is captured, it may be used to construct and/or merge pointclouds.

According to various embodiments, the capture of image data may be aidedby lights. Lights used in conjunction with an undercarriage imagingsystem may include LEDs, tube lights, flat sheet lights, or othersuitable lighting. Lights may be used in conjunction with reflectors,mirrors, or other such devices to shape the light intensity.

The stitched image is stored at 4916. According to various embodiments,stitched images may be stored on a local storage device. Alternatively,or additionally, stitched images may be transmitted to a remote machinevia a network.

FIG. 50 illustrates a method 5000 performed in accordance with one ormore embodiments. According to various embodiments, the method 5000 maybe performed on a computing device having a display screen and a userinput mechanism. For instance, the method 5000 may be performed on alaptop computer, a desktop computer, or a mobile phone.

A request to present MVIDMRs of a vehicle undercarriage is received at5002. In some implementations, the request may be generated as part of arequest to load a user interface for accessing visual data associatedwith a vehicle. For instance, the request may be generated in a nativeapplication or on a web browser loading a web page.

MVIDMRs from two or more camera perspectives are presented at 5004. Insome embodiments, each MVIDMR may include multiple perspectives of thevehicle undercarriage. In some MVIDMRs, each image may be captured fromthe same camera. Alternatively, a single MVIDMR may include imagescaptured by different cameras. For instance, images from a row or columnof cameras may be stitched together, and the stitched images used tocreate an MVIDMR.

FIGS. 51 and 52 show examples of presenting multiple MVIDMRs, providedin accordance with one or more embodiments. In FIGS. 51 and 52 , fourdifferent MVIDMRs of a vehicle undercarriage are presented, each beingcaptured from a different camera. In this way, the same portion of theundercarriage can be viewed from different angles. Because theundercarriage is three-dimensional, presenting the undercarriage fromdifferent perspectives allows seeing portions of the undercarriage fromone perspective that are occluded by an obstruction in anotherperspective.

FIG. 54 shows another example of presenting multiple MVIDMRs. In FIG. 54, four different MVIDMRs are presented, initially presented in a statejust prior to a vehicle driving over the cameras.

According to various embodiments, MVIDMRs captured from differentperspectives may be presented in a single view. In this way, user inputmay be employed to allow navigation of the MVIDMR in two or moredimensions. In a first dimension, the MVIDMR perspective may be shiftedfront-to-back and back-to-front to reflect the passage of the vehicleover the cameras. In a second dimension, the MVIDMR perspective may beshifted left-to-right and right-to-left to view image data captured fromdifferent cameras arranged across the vehicle undercarriage. In a thirddimension, the MVIDMR perspective may be zoomed in or out. In particularembodiments, navigating an undercarriage MVIDMR in two or moredimensions may allow a user to zoom in, zoom out, look at, and/or lookbehind components that are included in a vehicle's undercarriage.

User input navigating in one or more dimensions is received at 5006.According to various embodiments, the user input may be provided via anyof a variety of suitable user input devices. For example, on a mobilephone or tablet computer the user input may include swiping in adesignated direction with a finger on a touch screen. As anotherexample, on a desktop or laptop computer the user input may includedragging with a mouse cursor.

The visual display of each MVIDMR is updated at 5008 based on thereceived user input. According to various embodiments, the visualdisplay may be updated by moving each MVIDMR to a different image. Forinstance, dragging a mouse cursor downward may cause the vehicle'sundercarriage to appear to move up the display screen. That is, thevehicle would appear to be traveling in a forward direction in the imagesequence. In contrast, dragging the mouse cursor upward may cause thevehicle's undercarriage to appear to move down the display screen. Thatis, the vehicle would appear to be moving in reverse.

A determination is made at 5010 as to whether to receive additional userinput. According to various embodiments, additional user input maycontinue to be received so long as the MVIDMRs are being presented. Forinstance, additional user input may continue to be received so long asthe web browser or native application in which the MVIDMRs are displayedcontinue to be open.

1. A method comprising: capturing a plurality of images of anundercarriage of a vehicle, the images captured via one or more cameras,each camera being arranged in a respective fixed position, the imagesbeing captured as the vehicle moves over the one or more cameras;determining a designated point cloud based on the plurality of imagesvia a processor, the designated point cloud including a plurality ofpoints positioned in a virtual three-dimensional space, the plurality ofpoints including a first point and a second point; determining adesignated stitched image based on the designated point cloud via theprocessor, wherein the stitched image is determined by projecting thedesignated point cloud onto a virtual camera view and identifying aplurality of correspondence points and a plurality of pointcorrespondence transformations associating a first location of the firstpoint in a first image with a second location of the second point in asecond image; and storing the designated stitched image on a storagedevice.
 2. The method recited in claim 1, wherein the virtual cameraview is a two-dimensional plane in the virtual three-dimensional space.3. The method recited in claim 1, wherein the virtual camera view is apoint in the virtual three-dimensional space.
 4. The method recited inclaim 1, wherein determining the designated point cloud comprisesdetermining a plurality of point clouds.
 5. The method recited in claim4, wherein the one or more cameras includes a plurality of cameras, andwherein each of the point clouds corresponds with a respective one ofthe cameras.
 6. The method recited in claim 4, wherein each of the pointclouds corresponds with a respective one of the images.
 7. The methodrecited in claim 1, wherein the designated point cloud is determinedbased at least in part on depth information captured from one or moredepth sensors.
 8. The method recited in claim 1, wherein determining thestitched image comprises identifying a plurality of correspondencepoints, each of the correspondence points indicating a visual featureincluded in two or more of the plurality of images.
 9. The methodrecited in claim 8, wherein determining the stitched image furthercomprises identifying a plurality of point correspondencetransformations, each of the point correspondence transformationsidentifying a respective spatial relationship between the first locationof a designated one of the correspondence points in a first one of theimages and the second location of the designated correspondence point ina second one of the images.
 10. The method recited in claim 9, whereindetermining the stitched image further comprises determining an imagecorrespondence providing a mathematical transformation between the firstand second images in a two or three dimensional coordinate space basedon the plurality of correspondence transformations.
 11. The methodrecited in claim 10, wherein the image correspondence is a numericalaverage of one or more of the point correspondence transformations, thenumerical average determined based on a cost function.
 12. The methodrecited in claim 1, wherein the designated stitched image is one of aplurality of stitched images determined based on the designated pointcloud.
 13. The method recited in claim 1, the method further comprisinggenerating a multi-view interactive digital media representation thatincludes the plurality of stitched images, the multi-view interactivedigital media representation being navigable in one or more dimensions.14. The method recited in claim 13, wherein the multi-view interactivedigital media representation being navigable in three dimensions.
 15. Asystem comprising: one or more cameras configured to capture a pluralityof images of an undercarriage of a vehicle, each camera being arrangedin a respective fixed position, the images being captured as the vehiclemoves over the one or more cameras; a processor configured to determinea designated point cloud based on the plurality of images and determinea designated stitched image based on the designated point cloud, thedesignated point cloud including a plurality of points positioned in avirtual three-dimensional space, the plurality of points including afirst point and a second point, wherein the stitched image is determinedby projecting the designated point cloud onto a virtual camera view andidentifying a plurality of correspondence points and a plurality ofpoint correspondence transformations associating a first location of thefirst point in a first image with a second location of the second pointin a second image; and a storage device configured to store thedesignated stitched image.
 16. The system recited in claim 15, whereindetermining the stitched image comprises identifying a plurality ofcorrespondence points, each of the correspondence points indicating avisual feature included in two or more of the plurality of images,wherein determining the stitched image further comprises identifying aplurality of point correspondence transformations, each of the pointcorrespondence transformations identifying a respective spatialrelationship between a first location of a designated one of thecorrespondence points in a first one of the images and a second locationof the designated correspondence point in a second one of the images,wherein determining the stitched image further comprises determining animage correspondence providing a mathematical transformation between thefirst and second images in a two or three dimensional coordinate spacebased on the plurality of correspondence transformations, and whereinthe image correspondence is a numerical average of one or more of thepoint correspondence transformations, the numerical average determinedbased on a cost function.
 17. The system recited in claim 15, whereinthe processor is further operable to generate a multi-view interactivedigital media representation that includes the plurality of stitchedimages, the multi-view interactive digital media representation beingnavigable in one or more dimensions.
 18. The system recited in claim 15,wherein determining the designated point cloud comprises determining aplurality of point clouds, and wherein the one or more cameras includesa plurality of cameras, and wherein each of the point clouds correspondswith a respective one of the cameras.
 19. The system recited in claim 1,wherein the virtual camera view is a two-dimensional plane or a point inthe virtual three-dimensional space.
 20. One or more non-transitorycomputer readable media having instructions stored thereon forperforming a method, the method comprising: capturing a plurality ofimages of an undercarriage of a vehicle, the images captured via one ormore cameras, each camera being arranged in a respective fixed position,the images being captured as the vehicle moves over the one or morecameras; determining a designated point cloud based on the plurality ofimages via a processor, the designated point cloud including a pluralityof points positioned in a virtual three-dimensional space, the pluralityof points including a first point and a second point; determining adesignated stitched image based on the designated point cloud via theprocessor, wherein the stitched image is determined by projecting thedesignated point cloud onto a virtual camera view and identifying aplurality of correspondence points and a plurality of pointcorrespondence transformations associating a first location of the firstpoint in a first image with a second location of the second point in asecond image; and storing the designated stitched image on a storagedevice.